DynamicAdjust: Dynamic Resource Adjustment for Mitigating Skew in MapReduce
نویسندگان
چکیده
منابع مشابه
SkewTune in Action: Mitigating Skew in MapReduce Applications
We demonstrate SkewTune, a system that automatically mitigates skew in user-defined MapReduce programs and is a drop-in replacement for Hadoop. The demonstration has two parts. First, we demonstrate how SkewTune mitigates skew in real MapReduce applications at runtime by running a real application in a public cloud. Second, through an interactive graphical interface, we demonstrate the details ...
متن کاملHandling Data Skew in MapReduce
MapReduce systems have become popular for processing large data sets and are increasingly being used in e-science applications. In contrast to simple application scenarios like word count, e-science applications involve complex computations which pose new challenges to MapReduce systems. In particular, (a) the runtime complexity of the reducer task is typically high, and (b) scientific data is ...
متن کاملHandling partitioning skew in MapReduce using LEEN
MapReduce is emerging as a prominent tool for big data processing. Locality is a key feature in MapReduce that is extensively leveraged in dataintensive cloud system: it avoids network saturation when processing large amount of data by co-allocating computation and data storage — the map phase. However, our studies with Hadoop, a widely used MapReduce implementation, demonstrate that the presen...
متن کاملA Study of Skew in MapReduce Applications
This paper presents a study of skew — highly variable task runtimes — in MapReduce applications. We describe various causes and manifestations of skew as observed in real world Hadoop applications. Runtime task distributions from these applications demonstrate the presence and negative impact of skew on performance behavior. We discuss best practices recommended for avoiding such behavior and t...
متن کاملSharesSkew: An Algorithm to Handle Skew for Joins in MapReduce
In this paper, we investigate the problem of computing a multiway join in one round of MapReduce when the data may be skewed. We optimize on communication cost, i.e., the amount of data that is transferred from the mappers to the reducers. We identify join attributes values that appear very frequently, Heavy Hitters (HH). We distribute HH valued records to reducers avoiding skew by using an ada...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: IEICE Transactions on Information and Systems
سال: 2016
ISSN: 0916-8532,1745-1361
DOI: 10.1587/transinf.2015edl8255